{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# textgenrnn 1.1 Transfer Learning\n",
    "\n",
    "by [Max Woolf](http://minimaxir.com)\n",
    "\n",
    "*Max's open-source projects are supported by his [Patreon](https://www.patreon.com/minimaxir). If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro\n",
    "\n",
    "You can use textgenrnn for text transfer learnin by training on one text dataset, and then another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from textgenrnn import textgenrnn\n",
    "\n",
    "textgen = textgenrnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2,000 texts collected.\n",
      "Training new model w/ 2-layer, 128-cell LSTMs\n",
      "Training on 114,560 character sequences.\n",
      "Epoch 1/10\n",
      "895/895 [==============================] - 137s 153ms/step - loss: 3.0760\n",
      "Epoch 2/10\n",
      "895/895 [==============================] - 134s 150ms/step - loss: 2.2337\n",
      "Epoch 3/10\n",
      "895/895 [==============================] - 127s 142ms/step - loss: 1.9428\n",
      "Epoch 4/10\n",
      "895/895 [==============================] - 125s 139ms/step - loss: 1.7691\n",
      "Epoch 5/10\n",
      "895/895 [==============================] - 127s 141ms/step - loss: 1.6444\n",
      "Epoch 6/10\n",
      "895/895 [==============================] - 125s 140ms/step - loss: 1.5440\n",
      "Epoch 7/10\n",
      "895/895 [==============================] - 131s 147ms/step - loss: 1.4551\n",
      "Epoch 8/10\n",
      "895/895 [==============================] - 128s 143ms/step - loss: 1.3736\n",
      "Epoch 9/10\n",
      "895/895 [==============================] - 124s 139ms/step - loss: 1.3018\n",
      "Epoch 10/10\n",
      "895/895 [==============================] - 129s 145ms/step - loss: 1.2386\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "The White House Inter Trump Tower Than His Protection In Trump\n",
      "\n",
      "Senate I want the walk to the wall to the wall of the embarrass\n",
      "\n",
      "Trump Said He Was His Promised About There Were Will White House Report\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Trump the defended to call for he signed president in Trump-Russia probe\n",
      "\n",
      "Trump Has Suppressed From President Trump Tower\n",
      "\n",
      "When the don't Jussions ban on US post of agents to testify deal the doggo\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "With Trump ands sace under that then back\n",
      "\n",
      "G O O P B\n",
      "\n",
      "This corgo doin' with towar top. : Y eppanted in the White House!” Democrats Ormows We and Clain Goind in alstruck\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_path = \"../datasets/reddit_rarepuppers_politics_2000.txt\"\n",
    "\n",
    "textgen.train_from_file(file_path, new_model=True, num_epochs=10, gen_epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To transfer the learnings, train the `textgenrnn` instance on another dataset without specifying `new_model=True`. You should train the second dataset for fewer epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2,000 texts collected.\n",
      "Training on 83,491 character sequences.\n",
      "Epoch 1/5\n",
      "652/652 [==============================] - 94s 144ms/step - loss: 2.1313\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "The What Hacker Community Control Company Firefox Has And Minutes\n",
      "\n",
      "The What Had I Carter Community Community Project Project\n",
      "\n",
      "The Finar Comminate Contan and Really Part\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Symasive years of the HN: A Probeach calls immigrations and Introducing the Interned and Introduction in the Programmer We Side Concased Computered Some the Transer The Rulesses to Demand the Finds Ud the Interfack of Amail Project And Are Bebsite in the Uber And Almert Hacker Which Dropwer With N\n",
      "\n",
      "Bought a Man Natter Introducing The Signan and Intert to Drone Comment Carter\n",
      "\n",
      "The Facebook Congress of The Free You Server Caname Go Facebook Company and Wing find the Mays\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Hocime Change, Dents of hack upies prodaccB Gtange\n",
      "\n",
      "Why Motle to Futse Werler Predjy casech\n",
      "\n",
      "HN: Gath Truits Returf-Soon’s Mode might\n",
      "\n",
      "Epoch 2/5\n",
      "652/652 [==============================] - 94s 144ms/step - loss: 1.8538\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Show HN: What Hacker Not Money Marror Show Heatle Mac Work Startup for Marst Marst Destroyed to Bucker Backed a Bucker and the Internet to the ware in the in the will and to the project\n",
      "\n",
      "Show HN: A released and the programming to the to out source and the program\n",
      "\n",
      "Show HN: A for the distributing to the in the more for the startup for the distribute to the ware for the startup for the program\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Show HN: Why died a flicked for the likisters\n",
      "\n",
      "Google Microsoft\n",
      "\n",
      "Show HN: A in the to do use do for a tools the research about still passes\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Hozil things – Gnoan Sing\n",
      "\n",
      "Show HN: Moces – Annound Silizoncary for Dack Itacley\n",
      "\n",
      "Baninst SV Hasstes prevore\n",
      "\n",
      "Epoch 3/5\n",
      "652/652 [==============================] - 102s 157ms/step - loss: 1.6955\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Ask HN: What was down and the started and starter on the Internet and I starters\n",
      "\n",
      "Show HN: A programmer and started and a servers to lose to be and I don't company to and some to the way to be startup programming to be and the world and the solution to in the press and I servers to be in the started and source programming for the started and some to resigns\n",
      "\n",
      "The Startup Startups\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Than the NSA from Fullagre\n",
      "\n",
      "Google States at US Startup Computer Startup\n",
      "\n",
      "Google Windows for the we new passes to $4400 (2016)\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "The UPR Computer\n",
      "\n",
      "My It’s deal\n",
      "\n",
      "Pyzzeds tracked teaman blindurs\n",
      "\n",
      "Epoch 4/5\n",
      "652/652 [==============================] - 96s 147ms/step - loss: 1.5615\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "Ask HN: What are commits and the people and the people of the successful in the secret programming a servers for a the support for you when you when you what a security and security installial projects\n",
      "\n",
      "Show HN: What are some password for a servers\n",
      "\n",
      "What Happens the Store Company\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "Not Email Phone Repeatedly Engineers Show Hello Wish Comment Trans The The Future of Android Store Photol Engineers Sharter Bastes Comments\n",
      "\n",
      "Are is in the webpage design\n",
      "\n",
      "I was a selling a private and I three programming and the support\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Edward Predare Source Circusion the Web Seam\n",
      "\n",
      "Facius Swartus\n",
      "\n",
      "Starting I as who ise of our themers\n",
      "\n",
      "Epoch 5/5\n",
      "652/652 [==============================] - 98s 150ms/step - loss: 1.4464\n",
      "####################\n",
      "Temperature: 0.2\n",
      "####################\n",
      "The Firefox and Started and Started and Startup\n",
      "\n",
      "The Facebook Scaler Says How the Firefox\n",
      "\n",
      "The Firefox and Started Services PayPal\n",
      "\n",
      "####################\n",
      "Temperature: 0.5\n",
      "####################\n",
      "The Facebook Department to the Unterafor in the Oraction\n",
      "\n",
      "Stack Computer the Stole in the Google Computer Matter\n",
      "\n",
      "A Broupboed a steal of a from and marriew\n",
      "\n",
      "####################\n",
      "Temperature: 1.0\n",
      "####################\n",
      "Microsoft in my lost up on a Internet was users\n",
      "\n",
      "White Your Encine-of Day Catchers\n",
      "\n",
      "Most gimit is now a mad\n",
      "\n"
     ]
    }
   ],
   "source": [
    "file_path = \"../datasets/hacker_news_2000.txt\"\n",
    "\n",
    "textgen.train_from_file(file_path, num_epochs=5, gen_epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LICENSE\n",
    "\n",
    "MIT License\n",
    "\n",
    "Copyright (c) 2018 Max Woolf\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "of this software and associated documentation files (the \"Software\"), to deal\n",
    "in the Software without restriction, including without limitation the rights\n",
    "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "copies of the Software, and to permit persons to whom the Software is\n",
    "furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all\n",
    "copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
